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ABSTRACT 

The  U.S.  Army  is  currently  interested  in  developing  state-of-the-art  training  methods  that  leverage  technology 
based  on  established  and  emerging  immersive  mixed  and  virtual  environment  systems  employing  both  head 
mounted  and  spatially  immersive  display  technologies.  A  primary  motivation  for  utilizing  mixed  and  virtual 
environment  systems  is  that  they  create  cost-effective  simulations  of  relevant  military  challenges  while 
training  a  variety  of  skills  and  processes.  However,  in  a  sometimes  single-minded  effort  to  directly  advance 
training  technology,  the  area  of  performance  testing  and  assessment  has  often  been  overlooked.  There  are  a 
number  of  compelling  reasons  to  address  the  assessment  component  concurrently  with  efforts  to  build  better 
training  tools.  It  is  nearly  a  given  in  modern  psychology  that,  before  one  can  begin  to  focus  on  training  or 
enhancement  of  any  behavior,  one  needs  to  be  able  to  measure  and  understand  the  performance  that  needs  to 
be  trained.  This  requires  assessment  tools  that  have  demonstrated  reliability  and  validity  for  measuring 
criterion  performance  both  pre  and  post  training  implementation.  Without  good  assessment  metrics,  the 
measurement  of  training  effectiveness  is  compromised,  and  as  well,  the  ability  to  investigate  and  understand 
the  components  of  effective  performance  that  are  needed  to  drive  the  evolution  of  a  training  system  is  not 
possible.  To  address  this  need,  we  have  initiated  development  of  a  comprehensive,  standardized,  norm-based 
VR  cognitive  performance  assessment  test  (VRCPAT)  battery. 


1.0  OVERVIEW 

Cognitive  performance  testing  is  not  a  new  area  for  the  U.S.  Army.  The  Army  Alpha/Beta  intelligence  tests 
from  WWI  provide  a  historical  illustration  of  the  military’s  quest  for  standardized  performance  assessment  to 
better  guide  selection,  placement,  and  training  decisions.  Since  that  time,  the  military  has  routinely  employed 
a  wide  range  of  performance  assessment  methods  based  on  paper  tests  and  subjective  behavioral  rating  scales. 
As  well,  simulation  technology  has  often  been  used  to  assess  task  specific  performance  primarily  for  ground 
vehicle  and  aircraft  equipment  operation.  These  efforts  represent  both  extremes  of  the  assessment 
spectrum — ^basic  paper  tests/rating  scales  and  high  level  simulation  technology — for  the  measurement  of 
vastly  different  criterion  performance  (general  declarative  knowledge/implementation  vs.  specific  highly 
proceduralized  skills).  The  VRCPAT  is  envisioned  to  fill  the  middle  ground  between  these  two  poles  by 
creating  a  battery  of  VR-delivered  performance  tests  that  will  serve  to  generate  a  normative  database  for 
performance  evaluation  and  comparison.  VRCPAT  will  leverage  the  assets  that  are  available  via  the  use  of 
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VR  to  assess  cognitive  performance  that  is  required  for  success  in  the  military,  within  the  context  of 
functionally  relevant  scenarios.  Much  like  an  aircraft  simulator  serves  to  test  and  train  piloting  ability  under  a 
range  of  controllable  stimulus  conditions,  the  long-term  vision  of  the  VRCPAT  is  to  use  VR  simulation 
technology  to  create  a  battery  of  tests  that  assess  the  specific  cognitive  components  that  underlie  all  facets  of 
successful  military  performance.  Such  cognitive  components  include  attention,  spatial  ability,  memory, 
executive  functioning  and  a  host  of  higher-level  language  and  reasoning  abilities. 

The  first  phase  of  our  VRCPAT  effort  focuses  upon  the  development  of  a  prospective  memory  module 
(PMM)  delivered  within  the  context  of  a  VR  middle  eastern  city  scenario  (Figures  1-2).  A  variant  of  this 
scenario  has  been  used  in  a  HMD  based  VR  system  designed  for  treating  soldiers  experiencing  post  traumatic 
stress  disorder  [1],  (Figure  3).  The  scenario  can  also  be  used  within  the  FlatWorld  mixed  reality  system  [2] 
which  combines  spatially  immersive  displays  with  theatrical  staging  techniques  (Figure  4). 

Prospective  memory  is  widely  regarded  as  the  real-world  memory  ability  needed  to  store  and  hold  information 
in  one’s  mind  for  later  retrieval  as  required  to  successfully  initiate  and  carry  out  a  future  functional  task  [3]. 
This  is  conjectured  to  be  a  core  cognitive  component  ability  that  has  general  impact  across  all  areas  of  military 
competency.  Traditional  tests  of  memory  performance  have  primarily  relied  on  word  or  object  list  learning. 
Typically,  a  person  being  tested  with  these  tools  is  orally  presented  with  a  list  of  10-16  contextually  irrelevant 
words  (or  pictures  of  objects,  etc.)  and  asked  at  a  later  time  to  recall  them.  While  such  tests  may  provide  a 
crude  measure  of  verbal  memory  storage  that  has  some  usefulness  for  diagnosing  clinical  populations  on  the 
low  end  of  the  performance  spectrum,  they  have  little  value  for  predicting  the  dynamic  cognitive  performance 
ability  required  to  carry  out  a  mission  in  the  types  of  complex  environments  that  are  characteristic  of  military 
operations.  VR-based  simulation  technology  approaches  are  considered  to  be  the  future  alternative  for 
devising  cognitive  assessment  measures  that  will  have  better  ecological/predictive  validity  for  real-world 
performance. 


Figure  1:  Overhead  View  of  Virtuai  City. 


Figure  2:  User  View  inside  Humvee. 
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Figure  3:  VR  PTSD  System  in  iraq. 


Figure  4:  FiatWorid  Mixed  Reaiity  System. 


2.0  ASSESSMENT  PROCESS 

The  VRCPAT-PMM  process  consists  of  three  primary  phases  described  below  followed  by  a  debriefing. 

2.1  Acquisition  Phase 

Users  are  presented  with  10  pieces  of  language-based  information  to  be  learned,  without  any  context  for  what 
they  will  need  to  do  with  this  information.  The  information  stimuli  will  be  language  based  (i.e.,  a  black  Opal 
truck;  the  number  661;  a  dog;  a  man  with  white  shoes;  an  abandoned  mangled  bicycle;  bags  of  cement;  a  red 
door;  a  broken  window;  etc.).  The  acquisition  phase  will  initially  be  standardized  to  five  minutes.  At  the 
very  end  of  the  acquisition  phase,  users  then  will  be  asked  to  name  the  objects  that  they  studied  as  an 
assessment  of  initial  declarative  recall  memory. 

2.2  VR  Interface  and  Generic  Task  Training  Phase 

Following  the  acquisition  phase,  a  five-minute  “interface  training”  period  will  then  occur  in  which  users  will 
become  familiar  with  the  controls  of  the  game  pad  navigation  interface  and  head  mounted  display  within  a 
non-relevant  generic  VR  scenario.  Users  will  navigate  a  barren  city-like  environment  and  will  be  required  to 
visit  five  target  zones  that  are  marked  with  a  blue  numbered  overhanging  light  that  will  signify  the  number 
identity  of  the  zone  where  they  will  perform  interface  object  location  and  selection  actions  similar  to  what 
will  be  required  in  the  next  phase. 

2.3  Retrieval  Phase 

The  users  will  remove  the  HMD  and  be  given  a  top  down  map  of  the  virtual  environment  that  contains  the 
five  sequential  well-marked  target  zones  (blue  light  marked).  Users  will  then  be  instructed  that,  when  they 
next  put  on  the  HMD,  a  mission  will  begin.  The  user  will  then  put  on  the  HMD,  and  audio  instructions  will 
be  presented  regarding  the  task.  Users  will  be  told  that  they  will  need  to  go  to  each  target  zone  in  sequence 
(as  they  did  in  the  previous  phase),  and  at  each  zone,  two  of  the  items  that  they  had  memorized  previously 
will  be  present  somewhere  in  the  environment  from  that  vantage  point.  Upon  finding  the  items,  they  should 
align  the  cross  hairs  with  that  object  and  press  the  response  button  to  eliminate  or  “collect”  them.  Users  will 
have  one  minute  to  spend  within  each  target  zone  and  scan  for  the  relevant  memorized  target  items.  If  they 
find  the  target  items  in  less  than  the  one-minute  period,  they  can  move  on  to  the  next  zone,  at  which  time  they 
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will  have  only  one  minute  upon  arrival  to  acquire  those  items.  If  the  user  does  not  find  both  objects  in  a 
target  zone  by  the  time  that  the  one-minute  period  has  elapsed,  an  alarm  will  sound  and  a  voice  will  tell  the 
user  to  move  to  the  next  zone  and  seek  out  the  two  objects  located  there. 

2.4  Debriefing  Phase 

During  this  phase,  users  will  be  asked  to  recall  the  original  list  of  stimuli  and  at  which  target  zone  they  were 
found.  Following  this  measure  of  pure  recall,  users  could  also  be  tested  using  cued  recall  and  recognition 
assessment.  Finally,  an  open-ended  debriefing  interview  will  be  conducted  to  gather  subjective  information 
on  user  strategies  that  could  be  used  both  to  further  interpret  the  results  and  to  evolve  the  design  of  the  test. 

The  performance  measures  that  will  be  derived  from  this  test  include:  number  of  correct  hits,  false  hits,  time 
to  successfully  complete  per  target  zone,  time  to  complete  overall,  tracked  total  scanning  behavior 
effectiveness,  and  efficiency  of  scanning.  These  metrics  will  be  used  to  develop  test  norms  by  age  and  gender 
for  evaluation  and  comparison,  and  these  scores  will  be  correlated  with  demographic  and  other  performance 
tests  measures  administered. 


3.0  SYSTEM  TECHNOLOGY 

The  current  VRCPAT  PMM  application  is  designed  to  run  on  a  Pentium  4  notebook  computer  with  1  GB 
RAM  and  a  256  MB  DirectX  9  compatible  graphics  card.  The  user  navigates  through  the  scenario  using  a 
simple  USB  game  pad  device  while  wearing  an  800x600  resolution  head  mounted  display  (E-Magin  Z800). 
In  addition  to  HMD  presentation,  the  VRCPAT  PMM  can  be  configured  for  delivery  on  a  standard  LCD 
monitor  or  using  large,  spatially  immersive  displays  in  either  monoscopic  or  stereoscopic  modes.  The 
application  is  built  upon  ICT’s  FlatWorld  Simulation  Control  Architecture  (FSCA)  [4].  The  FSCA  facilitates 
a  network-centric  system  of  client  displays  driven  by  a  single  controller  application.  The  controller 
application  broadcasts  user-triggered  or  scripted-event  data  to  the  display  clients.  The  real-time  3D  scenes  are 
presented  using  Emergent’s  Gamebryo  graphics  engine.  Art  content  is  created,  edited  and  exported  to  the 
engine  using  Autodesk’s  Maya  software. 
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X  Overview 

*  Motivation 

*  Virtual  Reality  and  Psychology 

*  Test  Structure 

*  Development  Plan 

*  Software,  Hardware  Platform 

*  Target  Systems 


Motivation:  Bridging  a  Gap 

•  Assessment  methods:  Two  extremes 

•  Technological  low  end:  Paper  and  pencil  tests,  WWI  Alpha/Beta  Intelligence 
tests 

o  Often  used  for  selection  and  placement,  test  general  cognitive  skills 

•  Technological  high  end:  Vehicle  simulators,  test  highly  proceduralized  skills 
over  a  range  of  conditions 

•  Two  approaches  for  measuring  different  criteria  represent  extremes  of  the 
assessment  spectrums 

•  How  can  you  bridge  this  gap  and  update  the  technology  used  to  test  cognitive 
components  such  as  attention,  spatial  ability,  memory,  executive  functioning 
and  a  host  of  higher-level  language  and  reasoning  abilities? 
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Why?  These  components  are  all  critical  for  successful  military 
performance 


10/23/2007  use 


X  Solution:  VRC  R^TBatbeiy 

•  Create  a  Virtual  Reality  Cognitive  Performance  Assessment  Test  (VRCPAT) 
Battery 

•  VRCPAT  uses  immersive  simulation  environments  to  deliver  tests  that  assess 
the  specific  cognitive  components  that  underlie  all  facets  of  successful  military 
performance. 

•  VRCPAT  can  deliver  the  test  under  a  range  of  controllable  stimulus  conditions 
within  relevant  simulated  military  operational  contexts 


X  VRand  Psychology 

•  VR  mental  rotation  tests 

•  ADHD  assessment 

•  VR  pain  distraction 

•  Vietnam  PTSD  (Georgia  Tech,  Atlanta  VA,  Emory  1997) 

•  Iraq  PTSD  (onr  2004-Present) 
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^  VRand  Radiology 

•  Virtual  Vietnam  (1997) 


let 

wftrhsSr 


10/23/2007  use 


I  'y- 


^  VRand  Psychology 


•  Sparse  graphics 

•  Rich  sound  environment  compensated,  spawned  audio-vision 
correlation  study  (1999) 

•  Clinically  validated  by  VA,  Emory  study 


^  VRand  Psychology 


•  Virtual  Iraq  (DSC,  ONR  sponsored  2004-Present) 


\  VRand  Psychology 
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Sent  to  Iraq  for  feedback  from  combat  stress  team 
Uses  game  rendering  engine  (Emergent  Gamebryo) 

Art  assets  from  Full  Spectrum  Warrior  and  other  game  based 


sim  projects 


X  VRC  I^T:  Hist  Sbep 

•  Prospective  Memory  Module  (PMM) 

•  Why  memory? 

•  Prospective  memory  is  widely  regarded  as  the  real-world  memory  ability 
needed  to  store  and  hold  information  in  one’s  mind  for  later  retrieval  as 
required  to  successfully  initiate  and  carry  out  a  future  functional  task. 

•  Conjectured  to  be  a  core  cognitive  component  ability  that  has  general 
impact  across  all  areas  of  military  competency. 


Example:  Presence  patrols 


X  nospeclive  Memory  Module 

•  Traditional  tests  of  memory  performance  have  primarily  relied  on  word  or  object  list 


learning. 

•  Variant  of  the  scenario  is  being  used  in  Virtual  Iraq 

•  PMM  is  delivered  within  the  context  of  a  virtual  middle  eastern  city 


X  nospeclive  Memory  Module 

•  Four  Phases 

•  Acquisition 

•  Interface  Training 

•  Retrieval 

•  Debriefing 
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X  Acquisition  Phase 


•  Users  are  presented  with  10  pieces  of  language-based  information  to 
be  learned,  without  any  context  for  what  they  will  need  to  do  with  this 
information 

•  A  black  Opal  truck;  the  number  661 ;  a  dog;  a  man  with  white 
shoes;  an  abandoned  mangled  bicycle;  bags  of  cement,  etc. 

•  At  the  very  end  of  the  acquisition  phase,  users  then  will  be  asked  to 
name  the  objects  that  they  studied  as  an  assessment  of  initial 
declarative  recall  memory. 


X  InlBifece  liaining 

•  Users  become  familiar  with  the  controls  of  the  game  pad  navigation 
interface  and  head  mounted  display  within  a  non-relevant  generic  VR 
scenario. 

•  Users  will  navigate  a  barren  city-like  environment  to  visit  five  target 
zones  that  are  marked  with  a  blue  numbered  overhanging  light 

•  Number  signifies  the  number  identity  of  the  zone  where  they  will 
perform  interface  object  location  and  selection  actions  similar  to  what 
will  be  required  in  the  next  phase. 


X  Retrieval  Phase 


•  Users  told  to  go  to  each  target  zone  in  sequence  as  they  did  in  the  previous  phase 

•  Two  objects  memorized  previously  will  be  in  each  zone 

•  Upon  finding  items,  users  align  them  in  a  reticle  and  collect  by  pressing  a  button 

•  Users  have  one  minute  in  each  zone  to  collect  the  items.  Two  of  the  items  that  they  had 
memorized  previously  will  be  present  somewhere  in  the  environment  from  that  vantage 
point. 

•  Users  can  go  to  the  next  zone  after  collecting  the  items 


If  one  minute  passes  without  collecting  the  objects,  they  are  forced  to  go  to  the  next  zone 


[)ebriefing  phase 


•  Pure  recall  test:  Users  will  be  asked  to  recall  the  original  list  of 
stimuli  and  at  which  target  zone  they  were  found. 

o  Next  users  can  be  tested  using  cued  recall 

•  Open-ended  debriefing  interview:  Gather  subjective  information  on 
user  strategies  that  could  be  used  both  to  further  interpret  the  results 
and  to  evolve  the  design  of  the  test. 


X  Perfbimance  Measutes 


•  The  performance  measures  that  will  be  derived  from  this  test  include: 

o  Number  of  correct  hits 
o  False  hits 

o  Time  to  successfully  complete  per  target  zone 
o  Time  to  complete  overall 
o  Tracked  total  scanning  behavior 

•  Metrics  will  be  used  to  develop  test  norms  by  age  and  gender  for 
evaluation  and  comparison 

•  Scores  will  be  correlated  with  demographic  and  other  performance 
tests  measures  administered 


X  Development  Han 

•  Software  complete  late  July/early  August  2006 

•  Test  trials  in  August  2006 

•  Additional  cognitive  assessment  modules  developed 
in  2007 


Conduct  tests  in  mixed  reality  systems  spanning  the 
virtuality  continuum 


X  Haidwaie,  Softwaie  Platfoim 


•  Rendering:  Emergent's  Gamebryo  Rendering  Library 

•  MR  System  Support:  Video  see-through  AR  using  AR-TAG,  multi¬ 
channel  projection  displays,  passive/active  stereo,  auto  stereoscopic 
displays,  HMD's 

•  Tracking:  E-magin,  Inertiacube2,  Polhemus,  3rd-tech 

•  Audio:  Tracked  OpenAL  3D  audio,  headphones,  dozens  of  channels 

•  Muitisensory  FX:  Audio  tactile  effects,  DMX  protocol  for  fog,  strobe, 

,  heat 

V  * 
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spatially  Immensive  Displays 


spatially  Immensive  Displays 


^  HatiA/brid  :  V\Mde  Area  Mixed  Reality 


^  HatlAibrid  Ihban'feiiain  Module 


FlatWorld  based  Urban  Terrain  Module:  Ft.  Sill  Oklahoma 


I  MR  System:  Temperature  controlled,  user  tracked,  handheld  virtual  binos 

ict 


X  Ac  knowfedgemerils 

•  PTSD  Project  Sponsorship:  Office  of  Naval 
Research,  Medical  and  Biological  Science  and 
Technology  Division,  CDR  Russell  Schilling,  Phd, 
MSC,  USN 

•  Programming:  Joseph  Nunn,  Anton  Treskunov, 
Dexter  Allen,  Brian  Manson 


Art:  Matt  Liewer,  Brad  Newman,  Ernie  Eastland 


Ac  knowledgemenis 


•  The  project  described  was  developed  with  funds  of  the 
Department  of  the  Army  under  contract  number  DAAD  19-99- 
D-0046. 

•  The  project  or  effort  described  here  has  been  sponsored  by 
the  U.S.  Army  Research,  Development,  and  Engineering 
Command  (RDECOM).  Statements  and  opinions 
expressed  do  not  necessarily  reflect  the  position  or  the 
policy  of  the  United  States  Government,  and  no  official 
endorsement  should  be  inferred. 


